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(54) Video encoder 



(57) The present invention includes a video encoder 
optimized so as to be implemented using general pur- 
pose DSPs or as software executable on a general pur- 
pose microprocessor. The present invention includes a 
modified motion detector module (50) which classifies 
blocks in the currently processing image as moving or 
stationary. Blocks classified as moving are then proc- 
essed by an modified motion compensation module (52) 



which uses a center-biased orthogonal searching pro- 
cedure to match the block with a previously processed 
image. A zero block detector module (54) determines 
whether the block will generate a zero-valued block after 
being processed by a transform module 16 and a quan- 
tization module 20. If a zero-valued block is predicted, 
the transformation module 1 6 and the quantization mod- 
ule 20 are bypassed. 
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Description 

TECHNICAL FIELD OF THE INVENTION 

This Invention relates to image processing and in 
particular to a method and system lor improved encod- 
ing of video data 

BACKGROUND OF THE INVENTION 

Image compression and decompression are used 
in a wide range of applications including video confer- 
encing systems, video phones, and motion picture 
transmission. The conventional approach in these ap- 
plications has been to use dedicated hardware for video 
coding, (i.e., image compression and decompression). 
The use of dedicated hardware, typically digital signal 
processors (DSPs), is required because the video cod- 
ing process is computationally expensive and slow on 
general-purpose hardware. As a result, widespread use 
of these applications has been stymied by the costs as- 
sociated with deploying the specialized hardware re- 
quired to achieve good performance. It has been pre- 
dicted, however, that these applications, in particular 
video conferencing, will become desktop commodities 
in the next tew years. Improvements to the video coder 
and the video coding process are necessary to make 
this prediction a reality. 

A diagram illustrating a video coding process in ac- 
cordance with International Telecommunication Union 
(ITU) standard H.263, hereinafter referred to as the ITU 
H.263 standard, for video coding and encoding at very 
low bit-rates, such as at 28K bits per second, is shown 
in Fig. 1. 

The video encoder shown in Fig. 1 includes a color 
transform module 12, a motion detector module 14, a 
motion compensation module 16, a transform module 
18, a quantization module 20, and a coding module 22. 
Also included is a feedback module 24 which includes 
an inverse quantization module 30, an inverse transfor- 
mation module 28, and a frame reconstruction module 
26. 

A video decoder, as shown in Fig. 2, performs the 
reverse process of the video coder and includes a bit- 
stream decoding module 40, the inverse quantization 
module 30, the inverse transform module 26, an inverse 
motion compensation module 42, and the frame recon- 
struction module 26. 

In the video coding process, motion compensation 
performed by the motion compensation module 16 is the 
most time consuming phase. The transformation and 
quantization phases, performed by the transformation 
module 18 and the quantization module 20, respective- 
ly, are also expensive phases to perform. 

However, with processor speeds doubling every 
two years, it is possible for software-only solutions to 
attain good performance and quality and to lower the 
costs associated with applications which require video 



processing enough make image processing a commod- 
ity item in desktop computing environments. 

To overcome the computational requirements of the 
various stages, the video processing applications in pri- 
s or art systems employ dedicated DSPs to make the var- 
ious computationally expensive stages execute faster. 
Use of dedicated hardware is a weakness of current vid- 
eo conferencing systems. Designing new hardware as 
the video coding standards change and evolve is expen- 
se srve, time-consuming and substantially increases the 
cost of delivered systems. Not only do the high costs 
associated with dedicated hardware present a barrier 
against image processing applications becoming a 
desktop commodity solutions, but they also go against 
is the latest trend in hardware/software solution to use 
open systems. 

Thus, what is needed is a method and system to 
overcome the limitations and weaknesses of current vid- 
eo processing application implementations. In particu- 
20 lar, what is needed is a method and system tor video 
encoding which is computationally more efficient than 
those of the prior art and which are amenable to imple- 
mentation using low-cost general-purpose DSPs or soft- 
ware-only solutions. 

25 

SUMMARY OF THE INVENTION 

The present invention includes a video coding proc- 
ess which can be implemented on low-cost, general pur- 

30 pose DSPs or as a software only solution on a general 
purpose microprocessor to get acceptable perform- 
ance. The present invention modifies the overall video 
coding process and augments it with optimizations 
which reduce the overall video coding computation time 

os thereby making it viable to use the low-cost DSP solu- 
tions or a software-only solution, both of which permit 
acceptable performance on today's desktop CPUs. 

One aspect of the present invention includes opti- 
mizations of the motion detection stage. 

40 Another aspect of the present invention includes 
optimizations of the motion compensation stage. 

Yet another aspect of the present invention includes 
the addition of a zero block prediction stage. 

45 DESCRIPTION OF THE DRAWINGS 

These and other features of the invention that will 
be apparent to those skilled in the art from the following 
detailed description of the invention, taken together with 
so the accompanying drawings in which: 

Fig. 1 is a btock diagram depicting a prior art video 
encoder; 

S5 Fig. 2 shows a block diagram illustrating a prior art 
video decoder; 

Fig. 3 illustrates a first embodiment of a video en- 
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coder in accordance with the present invention; 

Fig. 4 depicts a second embodiment of a video en- 
coder in accordance with the present invention; 

Fig. 5 shows a third embodiment of a video encoder 
in accordance with the present invention; 

Fig. 6 is a flow diagram illustrating the operation of 
a video encoder in accordance with the present in- 
vention; 

Fig. 7 shows a flow diagram illustrating the opera- 
tion of a modified motion detector module in accord- 
ance with the present invention; 

Fig. 8 illustrates a flow diagram showing the oper- 
ation of a modified motion compensation module in 
accordance with the present invention; and 

Fig. 9 is a block diagram depicting the operation of 
a zero block predictor module in accordance with 
the present invention. 

DETAILED DESCRIPTION OF THE INVENTION 

A video coder in accordance with the present inven- 
tion, as shown in Fig. 3, includes several improvements 
to the video coder and video coding process of the prior 
art, shown in Fig. 1. The improvements shown in the 
video coder in accordance with the present invention as 
depicted in Fig. 3 provide for increased performance 
over video coders and video coding processes of the 
prior art. It is contemplated that the video coder and vid- 
eo coding process in accordance with the present in- 
vention are useful in any application using video coding 
techniques. 

The present invention is implemented in C++ using 
Microsoft's Visual C++ compiler on a workstation which 
included in a Pentium processor. It is contemplated, 
however, that use of other languages and hardware con- 
figurations would be apparent to those skilled in the art. 

The overall structure of the video coder in accord- 
ance with the present invention is shown in Fig. 3. As 
discussed hereinabove, the motion compensation stage 
and the transformation stage are among the most time 
consuming stages in a typical video coding process. The 
present invention provides for improvements to the mo- 
tion detection stage as performed by modified motion 
detector module 50. The modified motion detector mod- 
ule 50 In accordance with the present invention deter- 
mines whether to bypass the motion compensation 
stage performed by the modified motion compensation 
module 52. The modified motion compensation module 
52 also includes improvements to the motion compen- 
sation stage over that of the motion compensation mod- 
ule 16 of the prior art. 

The video coding process in accordance with the 



present invention also includes the addition of a zero 
block prediction stage, performed by zero block predic- 
tor module 54. The zero block predictor module 54 de- 
termines whether or not to bypass the transformation 

5 and quantization stages. Each of these features are dis- 
cussed in detail hereinbelow. 

It is contemplated that other embodiments, in addi- 
tion to that shown in Fig. 3, of the present invention are 
possible which also provide improvements to the video 

io coder of the prior art. Another embodiment of the video 
coder in accordance with the present invention is shown 
in Fig. 4, which includes the modified motion compen- 
sation module 52. Fig. 5 illustrates yet another embod- 
iment of video coder in accordance with the present in- 

is ventkxi which includes the modified motion detector 
module 50 and the zero block predictor module 54. 

The operation of the video encoder shown in Fig. 3 
is illustrated in Fig. 6. At block 60, the color transforma- 
tion module 12 generates data signals from the video 

20 image signals input at 51 . 

The data signals are then input to the modified mo- 
tion detector module 50 as shown at block 62 which de- 
termines whether movement is detected in the repre- 
sented image. If, at decision block 64, movement is de- 

25 tected, the modified motion compensation module 52, 
as shown at block 66, compensates for that movement 
and operation continues at block 68. If, at decision block 
64, movement Is not detected, the motion compensation 
stage performed by the modified motion compensation 

so module 52 is bypassed and operation continues at block 
66. 

At block 68, the zero block predictor module 54 de- 
termines whether the currently processing macrobbck 
of data signals will generate a zero-valued macrobbck 

35 after the transformation and quantization stages per- 
formed by the transform module 1 8 and the quantization 
module 20, respectively. If, at decision block 70, a zero- 
valued macroblock is predicted, the transformation and 
quantization stages are bypassed and operation contin- 

40 ues at bkxk 78. 

If, at decision bkxk 70, a zero-valued macroblock 
is not predicted, operation continues at block 72 where 
the data signals are transformed by the transform mod- 
ule 18. In the present invention, the transform module 

45 18 performs a discrete cosine transformation but, it is 
contemplated that other transformations may be used. 
The transformed data signals are then quantized at 
block 74 by the quantization module 20. At block 76, the 
data signals of the currently processed image are used 

so to generate an interpolated image which is fed back into 
the modified motion compensation module 52 at block 
66. 

The quantized data signals generated at block 74 
are encoded at block 78 for further processing by the 
55 application which incorporates or uses the video coder 
of the present inventbn. 

In the motion detection stage performed by the 
modified motion detection module 50, the operation of 
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which is illustrated in detail in Fig. 7, the macroblocks of 
the currently processing image processed are classified 
as either moving or stationary. In order to classify the 
macroblocks, the modified motion detection module 50 
determines if the macroblock has moved based upon 
predetermined motion criteria. The modified motion de- 
tector module 50 then, using the predetermined motion 
criteria, compares the currently processing macroblock 
with a macroblock which is in the same location in the 
previous image, the interpolated image generated by 
the feedback module 24, as the currently processing 
macroblock is in the current image. 

The pixel-by-pixel absolute difference between the 
macroblock in the currently processing image and the 
macroblock at the same location in the previous image 
is calculated. If the difference between a pixel in the cur- 
rently processing image and a pixel in the same location 
in the previous image is above a predetermined thresh- 
old, then that pixel is classified as moving within that 
macroblock. If the number of pixels classified as moving 
are above a second predetermined threshold, then the 
macroblock is classified as moving, otherwise the mac- 
roblock is classified as stationary. If the macroblock is 
stationary then the motion compensation phase is 
skipped entirely and computation proceeds as shown in 
Fig. 6. 

Returning to Fig. 7, if, at decision block 110, the 
modified motion detector module 50 is processing a 
macroblock in the first frame of data representing the 
current image (i.e., if it is an INTRA coded block), that 
macroblock is classified as stationary at block 1 34 and 
processing by the modified motion detector module 50 
terminates. If, at decision block 110, the current mac- 
roblock is not the first macroblock being processed, 
processing continues at block 112 where a counter and 
a maximum data signal indicator are initialized. Opera- 
tion then continues at block 114 where the first pixel in 
the currently processing macroblock is retrieved. 

If, at decision block 116, the signal value of the cur- 
rent pixel in the currently processing macroblock is 
greater then the current value of the maximum data sig- 
nal indicator, then the value of the maximum data signal 
indicator is replaced with the signal value at block 118 
and operation continues at block 120. If, at decision 
block 116, the signal value of the current pixel is not 
greater than the current value of the maximum data sig- 
nal indicator then operation continues at block 120 
where the modified motion detector module 50 deter- 
mines the difference between the signal value of the cur- 
rent pixel and the signal value of the pixel in the same 
location in the previous image. 

If, at decision block 122, that difference is greater 
than the first predetermined threshold, the counter is in- 
cremented at block 1 24 and operation continues at de- 
cision block 1 26. If, at decision block 1 22, the difference 
is not greater that the first predetermined threshold, op- 
eration also continues at decision block 126 where, if 
there are more pixels in the currently processing mac- 



roblock. the next pixel is retrieved at block 128 and op- 
eration continues at decision block 116. If, at decision 
block 1 26, all pixels in the current macroblock have been 
processed, a determination is made at decision block 

s 1 30 as to whether the counter is above the second pre- 
determined threshold. 

If, at decision block 130, the counter is above the 
second predetermined threshold, the macroblock is 
classified as moving at block 132. If, at decision block 

10 1 30, the counter is not above the second predetermined 
threshold, the macroblock is classified as stationary at 
block 134. Operation in the modified motion detector 
module 50 then terminates. 

Returning to Fig. 6, if, as shown at decision block 

is 64, the modified motion detector module 50 classifies 
the current macroblock as moving, the motion compen- 
sation stage performed by the modified motion compen- 
sation module 52 is not bypassed. The modified motion 
compensation module 52 operates as shown in Fig. 8. 

20 The motion compensation module 16 in video cod- 
ers of the prior art and the modified motion compensa- 
tion module 52 in accordance with the present invention 
both uses a search procedure to find a best matching 
macroblock in the previous image then both perform a 

25 half-pel search around the best matching macroblock in 
an interpolated version of the represented image. If a 
full search block matching procedure is used, as in the 
motion compensation module 16 in video coders of the 
prior art, the motion compensation phase is an order of 

30 magnitude more expensive (in terms of adds and multi- 
ples) than any other phase of the video coding process. 
Therefore, to reduce the complexity of the motion com- 
pensation stage, the modified motion compensation 
module 52 of the present invention includes several im- 

35 provements. 

As shown in block 80 in Fig. 8, the modified motion 
compensation module 80 first retrieves the macroblock 
from the previous image which is in the same location 
in the previous image as the currently processing mac- 

40 robkxk is in the current image. The modified motion 
compensation module 52 then determines, at block 82, 
the sum of the absolute difference between the retrieved 
macroblock from the previous image and the currently 
processing macroblock in the current image. 

45 if, at decision block 84, the sum of absolute differ- 
ence between the macroblock in the currently process- 
ing image and the macroblock at the same location in 
the previous image is below a certain predetermined 
threshold, then no further search is done and processing 

so continues at block 92 where the retrieved macroblock 
in the previous image is chosen as the best matching 
block. 

If, at decision block 84, the sum is above the pre- 
determined threshold, processing continues at block 86 
ss where eight neighboring points around the center of the 
retrieved macroblock in the previous image are deter- 
mined, the modified motion compensation module 52 
then determines, at block 88, the sum of the absolute 



4 



7 



EP 0 788 907 A2 



B 



difference between the pixels in the currently processing 
macroblock and each macroblock in the previous image 
centered around one of the eight neighboring points. 

This center biased orthogonal search algorithm (C- 
OSA) for block matching used by the motion compen- 
sation module 52 of the present invention is an optimi- 
zation of the orthogonal search algorithm (OSA) de- 
scribed in the article entitled "An Efficient Block-Match- 
ing Algorithm for Motion-Compensated Coding* by A. 
Puri, etal. and published in Proceeding IEEE ICASSSP 
1987 on pages 25.4.1-25.4.4. The C-OSA used in the 
modified motion compensation module 52 in the present 
invention performs a quick search on a set of eight 
neighboring points (+ or - one pixel in each direction and 
along the diagonals) centered around the search point 
in the retrieved macroblock in the previous image. 

In the center-biased search procedure used in the 
modified motion compensation module 52 of the present 
invention, if, at decision block 90, the best-match occurs 
at the center, then no further search is done and 
processing continues at block 92 where the macroblock 
centered around that best matching point is chosen. II, 
however, at decision block 90, the best match occurs at 
one of the eight neighboring points, processing contin- 
ues at decision block 94. 

If, at decision block 94, the best match is at an or- 
thogonal point ( a point which is + or - p pixel, where p 
is equal to 7, in any direction from the center point, i.e., 
not one of the points along a diagonal) then the search- 
ing procedure performed by the modified motion com- 
pensation module 52 in the present invention proceeds 
. as in the OSA procedure descrtoed in Puri, el al. If, how- 
ever, at decision block 94, the best match occurs instead 
at one of the eight neighboring points, then a new set of 
eight neighboring points is selected with that best 
matching point as the center point. The best match 
amongst this new set of eight neighboring points is re- 
turned at block 100 as the result. Note that in the new 
set of eight neighboring points, not all eight points have 
to be recomputed since some were already computed 
previously. 

It is worthwhile noting that the approach taken by 
the modified motion compensation module 52 favors 
motion compensation on images where there is relative- 
ly little motion between successive images, which is typ- 
ical of video conferencing applications. The C-OSA pro- 
cedure used by the modified motion compensation mod- 
ule 52 of the present invention includes a quick test to 
find the best matching block in a very small search area. 
In the worst case the modified motion compensation 
module 52 of the present invention does a full orthogo- 
nal search which is still less complex and produces good 
results. The complexity of the C-OSA used in the mod- 
ified motion compensation module 52 of the present in- 
vention is 1 6+4log2w, where w is search distance in pix- 
els, typically plus or minus 15 pixels. 

Finally, as shown at block 102, the modified motion 
compensation module 52 of the present invention ap- 



plies the OSA procedure described in Puri, et al. to the 
half-pel motion compensation search. The half -pel 
search is done on the interpolated image in a search 
area of plus or minus one pixel in all directions and along 

s the diagonals. Surprisingly, this took more than a sub- 
stantial proportion of the overall motion compensation 
time used by the modified motion compensation module 
52 since the full-pel search described hereinabove is al- 
ready quite efficient. Applying the OSA procedure de- 

io scribed in Puri, et al. on the half -pel search substantially 
reduced the number of search points. Also, the search 
area for border blocks is much smaller, since part of the 
area is not within the picture boundary, which provided 
additional savings in computation time. 

75 Results obtained from the C-OSA and OSA proce- 
dures applied to full-pel and half -pel search, respective- 
ly, by the modified motion compensation module 52 
were very close to an exhaustive search with no notice- 
able degradation of image quality and resulted in an or- 

20 der of magnitude speed up of motion compensation 
when compared to the exhaustive search. On several 
ITU H.263 standard test images, including the 'Miss 
America' image, motion compensation takes less than 
1 2% of the total computing time with less than a .01 dB 

25 decrease in signal-to-noise ratio. 

Returning to Fig. 6, on typical image sequences, e. 
g., on ITU test image sequences, less than 90% of IN- 
TER coded blocks had non-zero values after the quan- 
tization stage at 74 but all blocks had to go through the 

so expensive transformation stage at block 72. Further- 
more, the non-zero valued blocks had only one or two 
non-zero coefficients. To exploit the above observation 
a zero block prediction stage at 70 performed by the ze- 
ro block predictor module 54 is included after the motion 

3$ compensation stage at 66. 

The modified motion detector module 50. as de- 
scribed in the hereinabove, classifies macroblocks as 
either moving or stationary. The modified motion detec- 
tor module 50 also records the maximum signal value 

40 and the number of stationary pixels in the entire mac- 
roblock. Based on a quantization value which changes 
as processing progresses, this information is used by 
the zero block predictor module 50 to determine or pre- 
dict whether a macroblock will have all zero values after 

45 the transformation and quantization stages. If a zero- 
valued block is predicted, then the transformation and 
quantization stages, at 72 and 74, respectively, are 
skipped and a zero value block is generated. 

The operation of the zero block predictor module 54 

so is shown in Fig. 9. At block 140, the prediction is set to 
'no*. If, at decision block 142, a macroblock within the 
first frame is being processed, processing in the zero 
block predictor module 54 terminates. Otherwise, at 
block 144, the zero block predictor module 54 applies 

ss heuristics using the maximum signal value, the number 
of stationary pixels, and the current quantization value 
to determine if the current macroblock will generate a 
zero-valued block after the transformation and quantf- 
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zatkxi stages. If, at decision block 146, a zero-valued 
block is predicted, then the prediction is set to a y*s a at 
block 148 and processing in the zero block predictor 
module 54 terminates. 

The heuristics used by the zero block predictor 
module 54 are only applied to INTER coded blocks. On 
average this saves about 30% of the blocks from going 
through the transformation and quantization stages. 
However, because of the conservative nature of the 
heuristic used by the zero block predictor module 54, 
40% of the blocks which would have been zero valued 
blocks still went through the expensive transformation 
and quantization stages. It is contemplated, however, 
that the percentage of the blocks that are predicted as 
zero-valued blocks by the zero block predictor module 
54 can be increased as the heuristics used are im- 
proved. 

Before this optimization was applied, the transfor- 
mation and quantization stages consumed about 30% 
of the total processing time. This reduced to about 1 8% 
after the above optimizations were applied. It is contem- 
plated that the processing time can be further reduced 
if improved heuristics for predicting zero-valued blocks 
are applied. Tne overall video coding process in the 
present invention achieves about a 5-6 fold increase 
over the video coding process the ITU H.263 standard. 
Thus, the video coding process in accordance with the 
present invention is computationally more efficient and 
amenable to implementation using low-cost DSPs or 
software-only solutions. 

The techniques and concepts described in this in- 
vention were developed specifically for the video coding 
process used by video conferencing systems. However, 
it is contemplated that the present invention is also ap- 
plicable to any application which uses video coding 
techniques for image compression. 

Although the present invention and its advantages 
have been described in detail, it should be understood 
that various changes, substitutions and alterations can 
be made herein without departing from the spirit and 
scope of the invention. 
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tion detector module and a second set of data 
signals representing a previous image gener- 
ated by a feedback module for compensating 
for motion in said current image; 
a computational module responsive to said 
modified motion compensation module for gen- 
erating digitized data signals from said first set 
of data signals; 

said feedback module responsive to said digi- 
tized data signals for generating said second 
set of data signals for input to said motion com- 
pensation module; and 

a coding module responsive to said digitized 
data signals for generating encoded data sig- 
nals. 

The video encoder as claimed in Claim 1 , wherein 
said motion compensation module is capable of ap- 
plying a center-biased orthogonal search tech- 
nique. 

The video encoder as claimed in Claim 1 or Claim 
2, wherein said motion detector module is further 
responsive to a maximum value signal and a coun- 
ter signal indicative of how many of said pixels are 
classified as stationary. 

The video encoder as claimed in Claim 3, further 
comprising: a zero block predictor module respon- 
sive to first set of data signals, said maximum value 
signal and said counter signal for generating a zero 
block signal indicative of whether computation on 
said first set of data signals will generate non-zero 
data; and 

said computational module responsive to said 
zero block signal for generating said digitized data 
signals from said first set of data signals. 

t 

The system of any Claims 2 to 4, wherein said mo- 
tion compensation module is operable to perform a 
full pel search in accordance with said center-bi- 
ased orthogonal search technique. 



Claims 

1 . A video encoder comprising: 

a color transform module for receiving input vid- 
eo signals and for generating a first set of data 
signals representing a current image, including 
a plurality of pixels; 

a motion detector module coupled to said color 
transform module and responsive to said first 
set of data signals for generating a classifica- 
tion signal classifying said current image as 
moving or stationary; 

a motion compensation module responsive to 
said classification signal generated by said mo- 



45 



so 



56 



The system of any of Claims 2 to 4, wherein said 
motion compensation module is operable to per- 
form a half pel search in accordance with an orthog- 
onal search technique. 
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